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FINAL  REPORT 


GRANT  #:  N000149610472  and  N00014-99-0255  (two  phases  of  the  same 
Grant ) 

PRINCIPAL  INVESTIGATOR:  Dr.  Jiajie  Zhang 
INSTITUTION :  University  of  Texas  at  Houston 

GRANT  TITLE:  Toward  A  Cognitive  Theory  of  Direct  Interaction 
AWARD  PERIOD:  1  Jan  1996  -  31  Dec  1999. 

OBJECTIVE :  To  develop  a  cognitive  theory  of  direct  interaction  in  the 
context  of  dynamic  retargeting  tasks  of  F/A-18;  to  study  the  user 
interface  issues  of  F/A-18  cockpit;  to  develop  recommendations  of  user 
interface  design  for  F/A-18  cockpit. 

APPROACH :  We  conducted  extensive  representational  analyses  of  five 
isomorphic  navigation  instruments  (VOR,  ADF,  RMI,  747,  and  F/A-18)  for 
dynamic  retargeting  tasks  in  various  airplanes.  Based  on  these  analyses 
we  used  the  theory  of  distributed  cognition  to  predict  the  directness 
and  representational  efficiencies  of  the  five  instriiments .  We  performed 
several  sets  of  experiments  of  learning  and  retargeting  tasks  to  test 
the  predictions .  Based  on  the  representational  analyses  and  empirical 
results,  we  built  an  Act-R  model  for  a  subset  of  the  retargeting  tasks. 

Accomplishments : 


In  1996-1997,  we  started  the  project  by  using  Visual  Basic  to  program 
the  cockpit  instruments  and  experiment  software  in  Windows  95 
environment.  At  the  same  time,  we  carried  out  a  comprehensive 
representational  analysis  of  five  navigation  instruments  (VOR,  ADF, 

RMI,  747*  and  F/A-18) .  This  representational  analysis  predicted  the 
following  directness  of  the  five  instr\aments :  747  =  F/A-18  >  RMI  >  ADF 
>  VOR.  We  performed  the  first  set  of  experiments  that  tested  and 
confirmed  the  prediction.  This  set  of  experiments  was  based  on  a  simple 
task  (the  so-called  positioning  task)  in  which  the  subjects  used  one  of 
the  five  instruments  to  locate  the  position  of  an  airplane  from  the 
information  available  on  the  instr\iments . 

In  1997-1998,  based  on  the  initial  experimental  work  on  the  simple 
positioning  task  we  performed  in  1996-1997,  we  designed  and  carried  out 
several  sets  of  dynamic  retargeting  experiments  for  the  four  of  the 
five  instruments:  VOR,  ADF,  RMI,  F/A-18.  747  instrument  was  removed 
from  the  list  because  it  was  very  similar  to  the  F/A-18  instrument.  The 
miniature  cockpit  for  these  tasks  was  implemented  in  Microsoft  Visual 
Basic.  In  the  retargeting  tasks,  a  pilot  first  identified  his/her 
horizontal  position  by  using  one  of  the  four  sets  of  instruments.  Then 
the  pilot  was  told  that  an  event  at  a  different  location  needed 
immediate  attention  and  was  given  the  bearings  of  the  location  in 
relation  to  his  airplane.  The  pilot's  task  was  to  fly  the  airplane  to 
the  new  location  as  quickly  as  possible.  Our  first  hypothesis  is  that 
with  different  sets  of  instruments  the  difficulties  of  performing  the 
retargeting  tasks  are  different,  with  a  difficulty  order  of  F/A-18  < 

RMI  <  ADF  <  VOR.  This  difficulty  order  was  confirmed  by  the  results 
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from  all  sets  of  experiments.  The  first  set  of  retargeting  experiments 
focused  on  the  comparison  of  moving  map  and  fixed  map  conditions  (i.e., 
pilot-centered  and  station-centered)  with  different  sets  of 
instruments.  In  addition  to  the  predicted  difficulty  order  of  the 
instruments,  we  found  that  the  fixed  map  version  was  easier.  The 
second  set  of  retargeting  experiments  focused  on  the  orientations  of 
the  maps,  which  is  either  consistent  or  inconsistent  with  the  heading 
of  the  airplane.  In  addition  to  the  predicted  difficulty  order  of  the 
instruments,  we  found  that  the  consistent  version  was  easier.  The  third 
set  of  experiments  focused  on  the  transfer  from  one  set  of  instruments 
to  another  set.  For  example,  pilots  who  were  trained  on  VOR  method  were 
asked  to  perform  the  same  tasks  with  the  F/A-18  instruments.  The 
results  showed  that  there  was  an  instrument -independent  general 
transfer. 

Based  on  the  experiments  described  above,  we  implemented  an  ACT-R  model 
for  the  dynamical  navigation  tasks.  Our  testing  of  the  model  was 
focused  on  the  simple  positioning  task.  The  model's  learning  behavior 
matched  that  of  subjects:  they  both  showed  a  power  function  of 
practice . 

In  1998,  both  the  PI  and  the  Co-PI  moved  to  University  of  Texas-Houston 
from  The  Ohio  State  University.  After  moving- related  setup,  we 
performed  the  following  work. 

Based  on  the  experimental  work  we  performed  before  1998  at  Ohio  State 
University,  we  designed  a  new  set  of  dynamical  retargeting  task.  In 
this  new  retargeting  task,  a  pilot  first  identifies  his/her  horizontal 
position  by  using  one  of  the  four  sets  of  instruments  (VOR,  ADF,  RMI, 
F/A-18) .  Then  the  pilot  is  told  that  an  event  at  a  different  location 
needs  immediate  attention  and  is  given  the  bearings  of  the  location  in 
relation  to  his  airplane.  The  pilot's  task  is  to  fly  the  airplane  to 
the  new  location  as  quickly  as  possible.  Unlike  the  previous 
retargeting  experiments  in  which  pilots  filed  the  airplane  with  both 
instruments  and  visible  map  displays,  in  the  current  experiment  the 
pilots  had  to  do  complete  instrument  flying,  that  is,  the  only 
information  available  for  the  pilot  was  the  readings  of  a  specific 
instrument  (e.g.,  VOR).  This  is  an  extremely  challenging  task.  We  tried 
an  early  version  of  this  experiment  at  Ohio  State  but  could  not  do  it 
because  none  of  the  undergraduate  students  could  successfully  perform 
the  task.  In  the  new  experiment,  we  simplified  the  task  and  recruited 
subjects  from  the  pool  of  graduate  students  at  UT  Houston.  Each  subject 
spent  3  to  4  hours  to  perform  the  task.  We  run  15  subjects  for  each  of 
the  two  of  the  four  instriaments  (VOR,  RMI)  .  The  result  for  the  two 
conditions  was  very  clear:  the  VOR  condition  was  significantly  much 
harder  than  the  RMI  condition  both  for  the  base-line  training  task  and 
for  the  retargeting  task  afterwards. 

We  visited  the  Naval  Research  Lab  at  China  Lake  in  January,  1999  and 
established  collaborations  with  them.  They  created  and  ran  simplified 
version  of  our  experiments.  We  obtained  their  data.  Due  to  a  bug  in 
their  computer  program,  part  of  data  could  not  be  interpreted. 

Johnny  Chuah  successfully  defended  his  master's  thesis  in  Psychology  in 
1998,  and  he  remained  at  Ohio  State  to  finish  his  Ph.D.  with  another 
advisor  after  the  PI  and  Co-PI  moved  to  Houston.  Kevin  McGrory 


completed  his  master's  degree  in  computer  science  and  accepted  a  job 
offer  at  Lucent  Technology  in  1998. 

We  are  currently  preparing  two  more  manuscripts  to  publish  the  results 
obtained  from  the  research  supported  by  this  grant.  One  paper  will 
develop  the  cognitive  theory  of  direct  interaction  that  was  originally 
proposed  for  the  project.  We  have  collected  enough  data  to  support  our 
initial  framework  of  the  theory  and  provide  details  for  the 
modification  of  the  initial  theory.  We  plan  to  submit  this  paper  to 
Human-Computer  Interaction.  The  second  paper  will  be  focused  on  Act-R 
modeling.  We  have  developed  an  initial  model  for  the  navigation  task 
and  tested  it  for  the  simple  positioning  task.  We  have  done  initial 
analysis  of  the  task  structures  of  the  complex  retargeting  task  and 
have  enough  empirical  data  to  support  a  full  model.  We  plan  to  submit 
this  paper  to  Cognitive  Systems. 

Conclusions : 


This  project  uses  dynamic  retargeting  in  the  F/A-18  as  the  task  domain 
and  testbed  to  (a)  develop  a  cognitive  theory  of  direct  interaction 
that  will  explain  and  predict  the  basic  phenomena  of  direct  interaction 
and  will  provide  a  set  of  principles  that  specify  how  to  achieve  direct 
interaction;  (b)  develop  a  methodology  of  representational  analysis 
that  will  provide  a  set  of  procedures  for  the  analysis  and  comparison 
of  direct  and  indirect  systems;  (c)  develop  a  computational  model  that 
will  make  detailed  behavioral  predictions  on  interface  usability  and 
learnability  for  a  variety  of  interface  tasks;  (d)  perform  detailed 
representational  analyses  for  the  key  instruments  and  displays  used  for 
retargeting  tasks . 

The  studies  over  project  period  demonstrate  that  (a)  the  cognitive 
theory  of  direct  interaction  can  systematically  generate  theoretical 
predictions  that  can  be  tested  in  experiments;  (b)  the  methodology  of 
representational  analysis  is  a  powerful  and  systematic  tool  for  the 
analyses  of  the  real-time  retargeting  tasks;  (c)  the  Act-R  based 
computational  model  has  promising  results;  and  (d)  the  analyses  of  the 
cockpit  navigation  instruments  can  be  considered  as  recommendations  for 
the  design  of  future  instriiments . 

The  major  scientific  significance  is  the  development  of  a  cognitive 
theory  and  a  computational  model  of  direct  interaction.  The  major 
military  significance  is  the  redesign  of  F/A-18  cockpit  instruments 
such  that  they  provide  direct  interaction  with  the  pilots  who  can  spend 
their  limited  cognitive  resources  to  other  critical  tasks  (e.g., 
fighting) .  The  major  commercial  significance  is  the  development  of 
guidelines  of  direct  interaction  user  interfaces  that  can  help  the 
design  of  general  software. 

Patent  Information :  NONE 

Award  Information;  The  PI  (Jiajie  Zhang)  and  Co-PI  (Todd  R.  Johnson) 
were  promoted  to  Associate  Professor  from  Assistant  Professor.  Johnny 
Chuah,  a  graduate  student  fully  supported  by  this  grant,  won  a  Summer 
Cognitive  Science  Fellowship  from  The  Ohio  State  University.  Hongbin 
Wang,  another  graduate  student  partially  supported  by  this  grant,  won 
the  Marr  Prize  for  his  work  on  another  ONR  sponsored  project,  and  he 


was  offered  an  Assistant  Professor  position  in  the  Department  of  Health 
Informatics  at  University  of  Texas  at  Houston. 
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